[feature-wip](hplsql) Integrate hplsql into doris to support stored procedure #20776

luozenglin · 2023-06-14T02:57:43Z

Proposed changes

Issue Number: close #xxx

Support for running the hplsql from the command line to execute a SQL statement or HPL/SQL script.

$ cat ./sql/proceduer.sql
CREATE PROCEDURE print_select()
BEGIN
 select * from testdb.cost;
END;
CALL print_select();

$ sh fe/bin/hplsql.sh -f './sql/proceduer.sql'
Query executed successfully (41 ms)
100	tom	2012-01-10	200

$ sh fe/bin/hplsql.sh -e 'select * from testdb.cost;'
Query executed successfully (54 ms)
100	tom	2012-01-10	200

Support for running the hplsql with mysql client.

mysql> set enable_hplsql = true;
Query OK, 0 rows affected (0.02 sec)

mysql> delimiter //
mysql> CREATE PROCEDURE citycount (IN country VARCHAR(10), OUT cities INT)
    ->        BEGIN
    ->          SELECT COUNT(*) INTO cities FROM testdb.city
    ->          WHERE name = country;
    ->        END//
Query OK, 0 rows affected (0.07 sec)

mysql> delimiter ;
mysql>  declare cities int = 0;
Query OK, 0 rows affected (0.51 sec)

mysql>  call citycount('beijing', cities);
Query OK, 0 rows affected (0.81 sec)

mysql>  print cities;
Query OK, 0 rows affected (0.03 sec)
2

Reference http://mail.hplsql.org/doc

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-06-14T02:58:56Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In bin/hplsql.sh line 44:
    JAVA="$(which java)"
            ^---^ SC2230 (info): 'which' is non-standard. Use builtin 'command -v' instead.


In bin/hplsql.sh line 82:
java_version="$(
^----------^ SC2034 (warning): java_version appears unused. Verify use (or export if used externally).

For more information:
  https://www.shellcheck.net/wiki/SC2034 -- java_version appears unused. Veri...
  https://www.shellcheck.net/wiki/SC2230 -- 'which' is non-standard. Use buil...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:

----------
--- bin/hplsql.sh.orig
+++ bin/hplsql.sh
@@ -107,4 +107,3 @@
 fi
 
 ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.hplsql.Hplsql ${HELPER:+${HELPER}} ${OPT_VERSION:+${OPT_VERSION}} "$@" </dev/null
-
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

github-actions · 2023-06-14T03:01:31Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-06-14T13:11:57Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' found no issues.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:

----------
--- bin/hplsql.sh.orig
+++ bin/hplsql.sh
@@ -102,4 +102,3 @@
 fi
 
 ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.hplsql.Hplsql ${HELPER:+${HELPER}} ${OPT_VERSION:+${OPT_VERSION}} "$@" </dev/null
-
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

github-actions · 2023-06-14T13:14:07Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-06-14T17:27:48Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' found no issues.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:

----------
--- bin/hplsql.sh.orig
+++ bin/hplsql.sh
@@ -102,4 +102,3 @@
 fi
 
 ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.hplsql.Hplsql ${HELPER:+${HELPER}} ${OPT_VERSION:+${OPT_VERSION}} "$@" </dev/null
-
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

github-actions · 2023-06-14T17:31:02Z

clang-tidy review says "All clean, LGTM! 👍"

luozenglin · 2023-06-14T17:33:51Z

run buildall

github-actions · 2023-06-14T17:34:25Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' found no issues.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:

----------
--- bin/hplsql.sh.orig
+++ bin/hplsql.sh
@@ -102,4 +102,3 @@
 fi
 
 ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.hplsql.Hplsql ${HELPER:+${HELPER}} ${OPT_VERSION:+${OPT_VERSION}} "$@" </dev/null
-
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

github-actions · 2023-06-14T17:41:16Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-06-15T03:21:55Z

clang-tidy review says "All clean, LGTM! 👍"

luozenglin · 2023-06-15T06:12:06Z

run buildall

luozenglin · 2023-06-15T13:29:32Z

run buildall

github-actions · 2023-06-25T12:54:09Z

clang-tidy review says "All clean, LGTM! 👍"

luozenglin · 2023-06-25T12:55:53Z

run buildall

github-actions · 2023-06-30T03:23:54Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-06-30T05:09:12Z

clang-tidy review says "All clean, LGTM! 👍"

luozenglin · 2023-06-30T06:04:39Z

run buildall

luozenglin · 2023-07-11T05:47:40Z

run buildall

github-actions · 2023-07-11T05:51:01Z

clang-tidy review says "All clean, LGTM! 👍"

luozenglin · 2023-07-12T17:04:33Z

run buildall

github-actions · 2023-07-12T17:12:39Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-07-12T17:39:37Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 53.47 seconds
stream load tsv: 508 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.5 seconds inserted 10000000 Rows, about 338K ops/s
storage size: 17168889289 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230713013936_clickbench_pr_177553.html

github-actions · 2023-07-17T03:16:29Z

clang-tidy review says "All clean, LGTM! 👍"

luozenglin · 2023-07-17T03:46:24Z

run buildall

hello-stephen · 2023-07-17T04:19:43Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 54.09 seconds
stream load tsv: 543 seconds loaded 74807831229 Bytes, about 131 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.5 seconds inserted 10000000 Rows, about 338K ops/s
storage size: 17169703891 Bytes

luozenglin · 2023-07-18T05:46:55Z

run buildall

github-actions · 2023-07-18T05:54:31Z

clang-tidy review says "All clean, LGTM! 👍"

luozenglin · 2023-07-19T05:23:54Z

run buildall

github-actions · 2023-07-19T05:31:01Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-07-19T06:06:04Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 50.2 seconds
stream load tsv: 507 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json: 25 seconds loaded 2358488459 Bytes, about 89 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17167477439 Bytes

luozenglin · 2023-07-20T06:17:32Z

run buildall

github-actions · 2023-07-20T06:24:15Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-07-20T06:55:05Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.05 seconds
stream load tsv: 507 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17169806938 Bytes

luozenglin · 2023-07-20T12:20:27Z

run buildall

github-actions · 2023-07-20T12:29:02Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-07-20T13:04:09Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.86 seconds
stream load tsv: 502 seconds loaded 74807831229 Bytes, about 142 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17164833661 Bytes

luozenglin · 2023-07-21T08:42:27Z

run buildall

github-actions · 2023-07-21T08:48:36Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-07-21T09:21:31Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.07 seconds
stream load tsv: 503 seconds loaded 74807831229 Bytes, about 141 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17170636572 Bytes

github-actions · 2024-01-18T00:09:08Z

We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!

luozenglin marked this pull request as draft June 14, 2023 03:30

luozenglin force-pushed the hplsql branch from 7405a07 to 569c797 Compare June 14, 2023 17:33

luozenglin marked this pull request as ready for review June 14, 2023 17:33

luozenglin force-pushed the hplsql branch from 36056a7 to 0dbf5b3 Compare June 25, 2023 12:48

luozenglin force-pushed the hplsql branch from 0dbf5b3 to ca9dee1 Compare June 30, 2023 03:15

luozenglin force-pushed the hplsql branch from 14ad49b to 8fc045f Compare July 11, 2023 05:43

github-actions bot added the kind/test label Jul 11, 2023

luozenglin force-pushed the hplsql branch from 8fc045f to e557d46 Compare July 12, 2023 17:04

luozenglin force-pushed the hplsql branch from 4f1fe02 to 19f3092 Compare July 18, 2023 05:46

luozenglin force-pushed the hplsql branch from 19f3092 to 0a6bf7f Compare July 19, 2023 05:23

luozenglin force-pushed the hplsql branch from 0a6bf7f to c899690 Compare July 20, 2023 06:17

luozenglin added 4 commits July 20, 2023 20:14

[feature-wip](hplsql) Integrate HPL/SQL into doris

c1997df

update

76dfa2b

update

b786410

fix cursor and add test

177838b

luozenglin force-pushed the hplsql branch from c899690 to 177838b Compare July 20, 2023 12:20

update

cc6e3e0

github-actions bot added the Stale label Jan 18, 2024

github-actions bot closed this Jan 19, 2024

xinyiZzz mentioned this pull request Mar 20, 2024

[feature-wip](plsql)(step1) Support PL-SQL #30817

Merged

[feature-wip](hplsql) Integrate hplsql into doris to support stored procedure #20776

[feature-wip](hplsql) Integrate hplsql into doris to support stored procedure #20776

Uh oh!

Conversation

luozenglin commented Jun 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

github-actions bot commented Jun 14, 2023

sh-checker report

Uh oh!

github-actions bot commented Jun 14, 2023

Uh oh!

github-actions bot commented Jun 14, 2023

sh-checker report

Uh oh!

github-actions bot commented Jun 14, 2023

Uh oh!

github-actions bot commented Jun 14, 2023

sh-checker report

Uh oh!

github-actions bot commented Jun 14, 2023

Uh oh!

luozenglin commented Jun 14, 2023

Uh oh!

github-actions bot commented Jun 14, 2023

sh-checker report

Uh oh!

github-actions bot commented Jun 14, 2023

Uh oh!

github-actions bot commented Jun 15, 2023

Uh oh!

luozenglin commented Jun 15, 2023

Uh oh!

luozenglin commented Jun 15, 2023

Uh oh!

github-actions bot commented Jun 25, 2023

Uh oh!

luozenglin commented Jun 25, 2023

Uh oh!

github-actions bot commented Jun 30, 2023

Uh oh!

github-actions bot commented Jun 30, 2023

Uh oh!

luozenglin commented Jun 30, 2023

Uh oh!

luozenglin commented Jul 11, 2023

Uh oh!

github-actions bot commented Jul 11, 2023

Uh oh!

luozenglin commented Jul 12, 2023

Uh oh!

github-actions bot commented Jul 12, 2023

Uh oh!

hello-stephen commented Jul 12, 2023

Uh oh!

github-actions bot commented Jul 17, 2023

Uh oh!

luozenglin commented Jul 17, 2023

Uh oh!

hello-stephen commented Jul 17, 2023

Uh oh!

luozenglin commented Jul 18, 2023

Uh oh!

github-actions bot commented Jul 18, 2023

Uh oh!

luozenglin commented Jul 19, 2023

Uh oh!

github-actions bot commented Jul 19, 2023

Uh oh!

hello-stephen commented Jul 19, 2023

Uh oh!

luozenglin commented Jul 20, 2023

Uh oh!

github-actions bot commented Jul 20, 2023

Uh oh!

hello-stephen commented Jul 20, 2023

Uh oh!

luozenglin commented Jul 20, 2023

luozenglin commented Jun 14, 2023 •

edited

Loading

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`